Skip to content

Thrall embedding - #20

Draft
tonytw1 wants to merge 145 commits into
libvips-croppingfrom
thrall-embedding
Draft

Thrall embedding#20
tonytw1 wants to merge 145 commits into
libvips-croppingfrom
thrall-embedding

Conversation

@tonytw1

@tonytw1 tonytw1 commented Jul 10, 2026

Copy link
Copy Markdown
Member

What does this change?

Embedding runs without the embedding lambda using ImageLoader and Thrall.

Tweaks to the general optionally of the embedder so that the absence of an embedding implementation gracefully ignores the embedding related code paths.

Generalises the embedding implementation then
migrates the embedding implementation from Bedrock to Gemini Embedding 2 for running in Google Cloud.

Embedding implementations define an embeddingSourceImageFormat to indicate what format it needs it's input images to be in.

ImageOperations defines a new createEmbeddingSource to produce this image format.

Image Loader calls createEmbeddingSource during the image upload process placing the formatted image in a new embedding sources S3 Bucket.

Image loader continues to publish the SQS embedding messages but Thrall is now the receiver.

Thrall gets a new SQS listener to consume the embedding messages, call the embedding API and publish the currentUpdateEmbedding messages onto the low priority queue for indexing.

Stores calculated embeddings in an S3 bucket so that they can be reused when reindexing.

Nudges More Like this to be more of a boolean filter and enables it for all images which have an embedding.

Requires this mapping patch:

curl -X PUT "https://<ES_HOST>:9200/<INDEX_NAME>/_mapping" \
    -H "Content-Type: application/json" \
    -d '{
      "properties": {
        "embedding": {
          "dynamic": "strict",
          "properties": {
            "geminiEmbedding2": {
              "dynamic": "strict",
              "properties": {
                "image": {
                  "type": "dense_vector",
                  "dims": 768,
                  "index": true,
                  "similarity": "cosine",
                  "index_options": {
                    "type": "int8_hnsw",
                    "m": 16,
                    "ef_construction": 100
                  }
                }
              }
            }
          }
        }
      }
    }'

How should a reviewer test this change?

How can success be measured?

Who should look at this?

Tested? Documented?

  • locally by committer
  • locally by Guardian reviewer
  • on the Guardian's TEST environment
  • relevant documentation added or amended (if needed)

@tonytw1
tonytw1 force-pushed the thrall-embedding branch 2 times, most recently from 3f685af to 4024c56 Compare July 12, 2026 11:43
@tonytw1
tonytw1 force-pushed the libvips-cropping branch 3 times, most recently from 4ba0933 to 80b4976 Compare July 12, 2026 21:36
@tonytw1
tonytw1 force-pushed the thrall-embedding branch 2 times, most recently from 3f9064e to 11cca56 Compare July 12, 2026 22:05
@tonytw1
tonytw1 force-pushed the libvips-cropping branch from 80b4976 to 5a33669 Compare July 13, 2026 21:06
@tonytw1
tonytw1 force-pushed the thrall-embedding branch 8 times, most recently from 711acd9 to 7ce6b8e Compare July 19, 2026 12:07
@tonytw1
tonytw1 force-pushed the libvips-cropping branch from 5a33669 to 79be6a6 Compare July 19, 2026 12:11
@tonytw1
tonytw1 force-pushed the thrall-embedding branch from 7ce6b8e to 962c3b3 Compare July 19, 2026 12:13
@tonytw1
tonytw1 force-pushed the libvips-cropping branch from 79be6a6 to 5184ad9 Compare July 23, 2026 21:08
@tonytw1
tonytw1 force-pushed the thrall-embedding branch 2 times, most recently from 2c6f748 to 78f709f Compare July 24, 2026 21:42
@tonytw1
tonytw1 force-pushed the libvips-cropping branch from 5184ad9 to f393db1 Compare July 25, 2026 20:04
@tonytw1
tonytw1 force-pushed the thrall-embedding branch 4 times, most recently from 5c1b666 to 9ba87a4 Compare July 26, 2026 12:14
@tonytw1
tonytw1 force-pushed the libvips-cropping branch from f393db1 to f5d8b51 Compare July 27, 2026 20:19
@tonytw1
tonytw1 force-pushed the thrall-embedding branch 2 times, most recently from 7253b10 to a0e2e3b Compare July 27, 2026 20:33
@tonytw1
tonytw1 force-pushed the libvips-cropping branch from f5d8b51 to e76e7ca Compare July 27, 2026 20:52
When the user pages left or right, the new image item becomes available instantly on the template.
However, the imgops optimisedImage URL is not directly available on the image item seen by the template.

The image item optimisedImage field is a template with placeholders which can be used to build a full imgops URL.
When the image item changes after a page change, the controller uses getFullScreenUri function to calculare an imgops URL
and passes it to the template on the optimisiedImage observable

To prevent the user seeing an inconsistent view of the new images description overlaid over the previous image,
the controller and template instantly hide the image using the ctrl.loading flag

The controller watches for the optimisiedImage observable to emit the updated imgops URL before releasing the ctrl.loading flag to reveal the image.

The getFullScreenUri call does not need to be debounced because it's not actually loading the optimised image;
it's only calculating the URL and is operation is probably zero cost and nearly instant.

This nearly instance return, coupled with the debounce (which would make sense if we were loading the image)
probably contributes to a race condition (which I don't fully understand)
whereby the optimisedImage observable never emits an event and the ctrl.loading flag never resets.

The updated image is never revealed and the preview screen appears to stick when paging.

To complicate the situation, the entire RxJS graph driving this is plugged into the regular polling for new images on the main Grid view
which happens every 15 seconds.

When the preview UI gets stuck it will be nudged within at least 15 seconds by this polling and will reveal the new image.

Because 10-15s has traditionally been the same order of magnitude for a particularly slow impops load, this bug has been assumed to be an imgops issue.

It is not imgops related; imgops is never actually called from the stuck page.
…select check box is clobbered by click through to single image screen.
…selects from preview.

Attach the onImageClick behaviour from results so the entire image toggles the selection when in selection mode
and the click is handled by the range selection aware code.
Label; Download master crop
HEIC is actually a subset of HEIF.
To better accommodates 2 landscape ratios.
…initial attempt; not the total number of attempts.
Elasticsearch hostname works in cloudbuild tests.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant